Apparatus and method for adaptive 3D noise reduction

ABSTRACT

A non-iterative 3D processing method and system is disclosed for generic noise reduction. The 3D noise reducer is based on a simple conversion of the five types of noise to equivalent additive noise of varying statistics. The proposed technique comprises also an efficient temporal filtering technique which combines Minimization of Output Noise Variance (MNV) and Embedded Motion Estimation (EME). The proposed temporal filtering technique may be furthermore combined with classical motion estimation and motion compensation for more efficient noise reducer. The proposed technique comprises also a spatial noise reducer which combines Minimum Mean Squared Error (MMSE) with robust and effective shape adaptive windowing (SAW) is utilized for smoothing random noise in the whole image, particularly for edge regions. Another modification to MMSE is also introduced for handling banding effect for eventual excessive filtering in slowly varying regions.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority under 35USC§119(e) of U.S. provisionalpatent application 60/592,339, entitled “Apparatus and method foradaptive 3D artifact reducing for encoded image signal” and filed Jul.30, 2004, the specification of which is hereby incorporated byreference.

TECHNICAL FIELD

The invention relates to image 3D noise reduction techniques primarilyoperable in real-time in an image or a sequence of images. Moreparticularly, the invention relates to adaptive spatio-temporalfiltering techniques suitable for many type of noise in imageapplications.

BACKGROUND OF THE INVENTION

The existing literature and/or patents on noise reducing techniques areabundant. Image de-noising techniques may be classified as spatial ortemporal ones or a combination of them. Spatial techniques relategenerally to some coring techniques applied to high frequency part of aconsidered image. Temporal de-noising techniques relate to temporalcoring techniques applied mainly in detected or estimated still parts ofa picture. Image de-noising techniques may be classified as spatial ortemporal ones. A series combination of the spatial and temporaltechniques, easier to do than a true 3D processing, is possible and maybe of advantage. In the following, the general trend of the subject willbe reviewed and the specific spatial or temporal exiting noise reducerswill be considered in some details.

The spatial noise reducing techniques may be applied to either stillpictures or to a sequence of images. In general, the spatial noisereducing techniques may be divided further into three categories.

In the first category, the spatial nonlinear filters are based on localorder statistics. These techniques may be found, for example, in A. R.Weeks Jr., “Fundamentals of Electronic Image Processing”, SPIE OpticalEngineering Press, Bellingham, Wash., 1996 or I. Pitas, and A. N.Venetsapoulos, “Nonlinear Digital Filters: Principles and Applications”,Kluwer Academic Publishers, Boston, 1990. Using a local window around aconsidered pixel, these filters are working on this set of pixelsordered now from their minimum to their maximum values. The medianfilter, the min/max filter, the alpha-trimmed mean filter, and theirrespective variants may be classified in this category. These filterswork well for removing impulse like salt-and-pepper noise. For the smallamplitude noise these filters can blur some details or small edges.

In the second category, the coring techniques are applied in anotherdomain different from the original image spatial domain. The chosendomain depends partly on the noise nature. The U.S. Pat. No. 4,163,258uses the Walsh-Hadamard transform domain; meanwhile, the U.S. Pat. No.4,523,230 suggests some sub-band decomposition. Finally, thehomomorphism filter, working in logarithmic domain, is the classical onefor removing multiplicative noise and image shading from an image.

In the third category, the filters are locally adaptive and the noiseremoving capacity is varying from homogenous regions to edge regions.

The well-known filter in this category is the minimum mean square error(MMSE) filter proposed originally by J. S. Lee in “Digital imageenhancement and noise filtering by use of local statistics”, IEEE Trans.on PAMI-2, March 1980, pp. 165-168. The filtered pixel output isadditively composed of local mean value and a pondered difference of thenoisy pixel and the local mean intensity values. The optimum weight,which corresponds to a kind of coring technique, may be determined foradditive noise by the local variance ratio of the true clean image andthe noisy one. The minimum mean square error filter removes noise wellfor homogenous regions and reserves the image edges. However, the noiseessentially remains in edge or near edge regions. Moreover, the optimumweight is changing for other types of noise.

A relationship of Lee's filter and recent Anisotropic Diffusiontechniques may be shown by Y. Yu and S. T. Acton in “Speckle ReducingAnisotropic Diffusion”, IEEE Trans. on Image Processing, vol. 11,November 2002, pp. 1260-1270.

In P. Chan and J. S. Lim, “One dimensional processing for adaptive imagerestoration”, IEEE Trans. on ASSP-33, February 1985, pp. 117-126, thereis presented a method for noise reducing in edge regions. The authorshave proposed the use, in series, of four (4) one-dimensional minimummean square error filters respectively along 0°, 45°, 90° and 135°directions. The obtained results are impressive for large variancenoise. For small noise, the filter can blur however some image edges.Moreover, the noise variance output at each filter stage is to be costlyestimated.

For the same purpose, in J. S. Lee, “Digital image smoothing and theSigma filter”, Computer Vision, Graphics, and Image Processing-24, 1983,pp. 255-269, the author has proposed a Sigma filter. For noise removing,the filter calculates, in a local window of 5×5 dimensions, the meanvalue of similar pixel intensities to that of the central consideredpixel. For small noise, the Sigma filter works well, except small imagedetails and some pixels with sharp spot noise. For the latter, J. S. Leehas suggested also, in a heuristic manner, the use of immediate neighboraverage at the expense of some eventually blurred picture edges.

U.S. Pat. No. 4,573,070 discloses a Sigma filter for a 3×3 window. Theauthor has combined, in a single configuration, the Sigma filter, anorder statistic filter and a strong impulse noise reduction filter.

In U.S. Pat. No. 6,633,683, a Shape adaptive Windowing combined bothminimum mean square error and Sigma Filter techniques, is disclosed.However, introduced banding artifact effect in slowly varying regionsand generic minimum mean square error structure for some usual types ofnoise are not considered.

The temporal filter is generally applied for a sequence of images inwhich the noise component is supposed to be non-correlated between twoor many successive images. The temporal filtering techniques are basedessentially on motion detection (MD) or motion compensation (MC). Thefilter structure may be IIR (infinite impulse response) or FIR (finiteimpulse response) filter with frame delay elements. In general, thetemporal techniques perform better than spatial ones. The system cost isdue essentially to the frame memory and the motion estimation. Thetemporal de-noising techniques may be found, for example, in U.S. Pat.Nos. 5,161,018, 5,191,419, 5,260,775, 5,404,179, 5,442,407, 6,061,100and in G. Wischerman, “The Digital Wetgate: A Third-Generation NoiseReducer”, SMPTE Journal, February 1996, pp. 95-100.

From a theoretical standpoint, a class of these noise filteringtechniques based on well established M C Kalman filtering is proposedfor spatio-temporal domain in Kim and Woods, “Spatio-temporal adaptive3-D Kalman filter for Video”, IEEE Transactions on Image Processing,Vol. 6, No. 3, March 1997. However, 3D Kalman filter is not convenientfor high speed implementation or abrupt scene change. Katsaggelos andal. in “Adaptive Image Sequence Noise Filtering Methods”, SPIE Vol. 1606Visual Communication and Image Processing 1991, pp 716-727, haveproposed two approaches for non stationary filtering of image sequences:a separable adaptive recursive motion compensated filter composed ofthree coupled 1-D estimators and a temporal non-linear filteringapproach without motion estimation. M. K. Ozkan et al. in “AdaptiveMotion Compensated Filtering of Noisy Image Sequences”, IEEE Trans. onCircuit and Systems for Video Technology, Vol. 3, No. 4, Aug. 1993, pp277-290 have suggested the use of adaptive weighted averaging filter forclaiming to overcome presence of edge, inaccurate motion estimation andscene change. Boo and Bose in “A motion-compensated spatio-temporalfilter for image sequences with signal dependent noise”, IEEE Trans. onCircuit and Systems for Video Technology, Vol. 8, No. 3, June 1998, pp287-298 have proposed a MC spatio-temporal filter using groups of frameand LMMSE in a transform domain.

The most interesting for the present invention is the second approach ofKatsaggelos and al.: a temporal non-linear filtering approach withoutexplicit motion detection or estimation. However, their approach iscostly in implying five frame memories and an inversion of matrix.

U.S. Patent Application No. 2001/0019633 discloses using a kurtosis ofthe noise as a metric for estimating the type of noise and appliedeither median filter or spatio-temporal filter in function of noisediscrimination.

SUMMARY OF THE INVENTION

The present invention provides an apparatus and method for efficientlyreducing noise in image signal.

According to an aspect of the present invention, there is provided anapparatus and method for reducing at least five (5) types of noise. Thefive types of considered noise are: a)—additive noise, b)—multiplicativenoise with photographic density gamma γ negative, c)—multiplicativenoise with photographic density gamma γ positive, d)—speckle or combinednoise of additive and multiplicative (γ<0) and e)—speckle or combinednoise of additive and multiplicative (γ>0) The apparatus and methodcomprises a noise power converter to convert said five various noisetypes into an equivalent, signal dependent, additive noise. When dealingwith an unknown noise type, the additive noise mode should be selected.

According to a further aspect of the present invention, there isprovided an apparatus and method for recursively temporal filtering. Inparticular, the temporal filtering introduces the criterion ofminimization of output noise variance (MNV) and the technique ofembedded motion estimation (EME). The former performs a noise reducingsuitable for high speed implementation. The latter provides efficienttechnique for overcoming presence of edge, inaccurate motion estimationand scene change.

According to a further aspect of the present invention, there isprovided an apparatus and method for recursively temporal filteringwhich is complementary with classical motion estimation andcompensation.

From another broad aspect of the present invention, there is provided anapparatus and method for spatial filtering which introduces shapeadaptive windowing (SAW) for efficient use of minimum mean square errortechnique in real life. Shape adaptive windowing is a robust again noiselocal classification of pixels in a window into two classes homogeneousor not in respect to a considered pixel.

According to a further aspect of the present invention, there isprovided an apparatus and method for spatial noise reduction which canhandle introduced banding effect artifact for eventual excessivefiltering in slowly varying image regions.

From another broad aspect of the present invention, there is alsoprovided an adaptive apparatus and method for noise reduction wheneverlocal noise power is known.

From another broad aspect of the present invention, there is alsoprovided an adaptive apparatus and method for noise reduction for thethree video components: luminance and two chrominance components.

The present description discloses an apparatus for reducing multiplenoise types in a video input signal, the apparatus comprising: a noisepower converter for receiving and using the video input signal, avariance of additive noise, a variance of multiplicative noise and anindication of a type of noise for estimating an equivalent additivenoise variance signal; a temporal recursive filter using the equivalentadditive noise variance and the video input signal for generating atemporally filtered video signal and a residual noise variance signal;and a spatial noise reducer using the residual noise variance signal andthe temporally filtered video signal for spatially filtering the videoinput signal to provide a video output signal having reduced noise.

The present description discloses a method for reducing multiple noisetypes in a video input signal, the method comprising: estimating anequivalent additive noise variance signal using the video input signal,a variance of additive noise, a variance of multiplicative noise and anindication of a type of noise; temporally filtering the video inputsignal video signal; generating a residual noise variance signal usingthe equivalent additive noise variance and the video input signal; andspatially filtering the temporally filtered video signal using theresidual noise variance signal and the video input signal to provide avideo output signal having reduced noise.

The present description discloses an apparatus for reducing multiplenoise types in a video input signal, the apparatus comprising: a noisepower converter for receiving and using the video input signal, avariance of additive noise, a variance of multiplicative noise and anindication of a type of noise for estimating an equivalent additivenoise variance signal; a spatial noise reducer using the equivalentadditive noise variance signal and the video input signal for spatiallyfiltering the video input signal to provide a video output signal havingreduced noise.

BRIEF DESCRIPTION OF THE DRAWINGS

Further features and advantages of the present invention will becomeapparent from the following detailed description, taken in combinationwith the appended drawings, in which:

FIG. 1 is a block diagram of a preferred embodiment of a multiple noisetype noise reduction (MTNR) apparatus;

FIG. 2 is block diagram of an embodiment of a noise power conversion offive given types of noise in accordance with the invention;

FIG. 3 is a block diagram of an embodiment of an embedded motionestimation temporal recursive filter;

FIG. 4 is a block diagram of an embodiment of an embedded motionestimation temporal recursive filter with classical motion compensation;

FIG. 5 is a block diagram of an embodiment of an embedded motionestimation temporal filter coefficient calculator;

FIG. 6 is a block diagram of an embodiment of a shape adaptive windowingspatial noise reducer;

FIG. 7 is a block diagram of an embodiment of an adaptive gain Kcalculator;

FIG. 8 is a block diagram of an embodiment of a region adaptive facetbased spatial noise reducer; and

FIG. 9 is a block diagram of another embodiment of a multiple noise typespatial noise reduction (MT-SNR) apparatus.

It will be noted that throughout the appended drawings, like featuresare identified by like reference numerals.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Now referring FIG. 1, there is shown an embodiment of a multiple noisetype noise reducing (MTNR) apparatus.

The multiple noise type noise reducing (MTNR) apparatus and method startwith two main system input information types. The first video inputinformation 101 is an image video signal composed of luminance Y andchrominance Cr, Cb components. Persons of ordinary skill in the art willunderstand that, except where indicated differently, such systemcomponents may be implemented in a time sharing manner or in parallel asit is well known in the art. The second information corresponds tocontrol parameters which are applied at input 102.

The control parameters signal at input 102 represent, for fiveconsidered types of noise, three additional types of information namely:noise type (number), variance of additive noise and variance ofmultiplicative noise. In the disclosed embodiment, this information isspecified by an end-user in a heuristic manner.

The multiple noise type noise reducing (MTNR) apparatus comprises anoise power converter (NPC) 104, an embedded motion estimation temporalrecursive filter (EME-TRF) 106 and a shape adaptive windowing spatialnoise reducer (SAW-SNR) 109.

The noise power converter (NPC) 104, described in detail below withreference to FIG. 2, receives the video input signal 101 and the controlparameters signal 102 and estimates equivalent additive noise localpower for each type of considered noise. The estimated local noise power(variance) 105 and the Noise Type number 110 are provided to theembedded motion estimation temporal recursive filter 106.

The embedded motion estimation temporal recursive filter (EME-TRF) 106receives the video input signal 101 and the local noise power signal 105to generate at its outputs 107 and 108 corresponding respectively to atemporally filtered video signal and a residual noise variance signal.The temporally filtered video signal and the residual noise variancesignal are provided in turn to the spatial noise reducer 109. Temporalrecursive filter techniques described in detail below with reference toFIGS. 3 and 4 are based on Embedded Motion Estimation (EME) andMinimization of residual Noise Variance (MNV).

The spatial noise reducer 109 described in details below with referenceto FIG. 8 receives the temporally filtered image signal 107 and thecorresponding residual noise variance signal 108 to perform a minimummean squared error filtering for spatial noise reduction with reducedbanding effect. The final resulting image (video output) is denoted by103.

It is worthwhile to mention that the temporal and spatial noisefiltering technique is based essentially on the knowledge of local noisevariance which can be either fixed or spatio-temporally varying.

Now referring to FIG. 2, there is illustrated in block diagram a noisepower converter for five considered types of noise in accordance withone embodiment of the invention.

The noise power converter (NPC) 104 receives the video input signal 101and the control parameters signal 102 and estimates the equivalent zeromean additive noise local power for each of the five types of considerednoise.

The five considered types of noise are additive (referred to also asN1), multiplicative with negative gamma γ photographic density (referredto also as N2), multiplicative with positive γ (referred to also as N3),speckle with γ negative (referred to also as N4) and speckle with γpositive (referred to also as N5).

The signal models and the proposed equivalent additive noise models ofthe five types of noise are summarized in table 1, in which:

f is the original image of local mean μ;

g is a noisy image;

u is a multiplicative noise of unitary mean and of σ² _(u) variance;

v is an additive noise of zero mean and of σ² _(v) variance;

and A is the amplitude signal value; (i.e. for an 8-bit representation,a possible value of A is 256).

It is assumed that the multiplicative noise u and the additive v noiseare statistically independent. σ² _(n) is the variance of the equivalentadditive noise.

TABLE 1 Noise Models and Equivalent Additive Noise Variance EstimatedEquivalent Equivalent Additive Type Signal Model Signal Model Noise σ²_(n) N1: Additive Noise g = f + v g = f + v σ² _(v) N2: Multiplicative g= f · u g = f · u μ² · σ² _(u) Noise (γ < 0) N3: Multiplicative A − g =g = A · (1 − (A − μ)² · Noise (γ > 0) (A − f) · u u) + f · u σ² _(u) N4:Combined or Speckle g = f · u + v g = f · u + v μ²σ² _(u) + Noise (γ <0) σ² _(v) N5: Combined or Speckle A − g = (A − g = A · (1 − (A − μ)² ·Noise (γ > 0) f) · u + v u) + f · u − v σ² _(u) + σ² _(v)

For example, in the N2 case, if g=f·u and g=f+n, where n is theequivalent additive noise then n=f·(u−1). Hence the true equivalentadditive noise variance is σ² _(n)=(μ²+σ² _(f))·σ² _(u). However, forsimplicity purpose and for direct optimization minimum mean square errorcompatibility the term σ² _(f) will be neglected and σ² _(n)=μ²·σ² _(u).Of course, μ is unknown and should be estimated with some accuracy.

In the N4 case, if g=f·u+v and g=f+n, where n is the equivalent additivenoise then n=f·(u−1)+v. Hence σ² _(n)=μ²σ² _(u)+σ² _(v).

The last column in previous table shows that the signal mean value μ isrequired for equivalent additive noise calculation. However, the signalmean value μ is unknown and must be estimated. The proposed estimationcan be done by a robust sliding Shape Adaptive Window as illustrated byFIG. 2.

The noisy luminance component Y is considered in the video input signal101 provided to low pass filter 201 and to mean estimator 206. The lowpass filter 201 is required for providing some robustness against noise.Its proposed impulse response is given as follows:

${{lp201}\left( {c,r} \right)} = {\begin{bmatrix}7 & 7 & 7 \\7 & 8 & 7 \\7 & 7 & 7\end{bmatrix}/64}$

It is a modified version without multiplier of the well-known box carfilter. (c,r) denotes the current pixel coordinates.

The output of the low pass filter 201 is applied now to a sliding localshape adaptive window (SAW) 202. The shape adaptive window (SAW) 202classifies, in turn, pixels in the window centered at the coordinates(c,r) into two categories: similar (“1”) or not (“0”) with the pixel ofinterest (c,r).

The following notation is defined:Y _(ij)(c,r)=Y(c+i, r+j).  (1)

The shape adaptive window 202 provides at its respective outputs 204 and205 the following signals:

$\begin{matrix}{{\omega_{ij}\left( {c,r} \right)} = \left\{ {{\begin{matrix}{{1,{{{if}\mspace{11mu}{{{{lp}\left( {Y_{ij}\left( {c,r} \right)} \right)} - {{lp}\left( {Y_{00}\left( {c,r} \right)} \right)}}}} < {Threshold}}}\;} \\{0,{else}}\end{matrix}{and}},} \right.} & (2) \\{{N\left( {c,r} \right)} = {\sum{\sum{{\omega_{ij}\left( {c,r} \right)}.}}}} & (3)\end{matrix}$

In which lp(Y(c,r)) is the low pass filter 201 output for luminancecomponent Y at (c,r), ω_(ij)(c,r) is a binary signal representing localclassification results within a threshold tolerance 203 and N(c,r) islocal similar pixels number in the window.

Equation (2) describes the robust local classification of the proposedshape adaptive window. One more time again, the low pass filter 201 isimportant for providing robustness against noise in imageclassification.

The shape adaptive window outputs 204 and 205 and the noisy luminanceinput signal 101 are applied together to shape adaptive window meanestimator 206 to provide a local estimated mean signal value μ_(Y)(c,r)output 207. The shape adaptive window mean estimator 206 performs thefollowing calculation:

$\begin{matrix}{{\mu_{Y}\left( {c,r} \right)} = {\left( {1/{N\left( {c,r} \right)}} \right){\sum\limits_{i}{\sum\limits_{j}{{Y_{ij}\left( {c,r} \right)}{\omega_{ij}\left( {c,r} \right)}}}}}} & (4)\end{matrix}$

The local estimated mean signal value μ_(Y)(c,r) output 207, theadditive noise variance σ² _(v) 209, the multiplicative noise varianceσ² _(u) 210, and the noise type signal 110 are applied together toequivalent additive noise variance calculator (EANVC) 208. In accordancewith the last column in the previous table, the equivalent additivenoise variance calculator 208 provides the variance σ² _(n) signal 105for the present consideration of luminance component, i.e. σ² _(n)=σ²_(Y). By default, noise will be considered by user as additive whennoise type is unknown. For skilled people in the art, there are manypossibilities, not shown in the Figures, for hardware or softwareimplementation of the required calculation.

Independent but similar calculations need to be carried out for each ofthe chrominance components Cr and Cb. However, for possiblesimplification, some consideration on segmentation results ω_(ij)(c,r)from luminance component may be foreseen. In fact, for 4:4:4 samplingpattern, the same results ω_(ij)(c,r) for luminance may be used even forboth Cr and Cb. For 4:2:2 sampling pattern, luminance classificationresults ω_(ij)(c,r) may be luminance co-sited sampled and hold, notshown, for both Cr and Cb before to apply to the shape adaptive windowmean estimator 206. In 4:4:4 R, G, B sampling pattern, localclassification ω_(ij)(c,r) are obtained independently for eachcomponent.

For each video component, the equivalent additive noise variancecalculator 208 yields corresponding variance results 105. That are σ²_(n)=σ² _(Y), σ² _(n)=σ² _(Cr) and σ² _(n)=σ² _(Cb) respectively forluminance, chrominance Cr and chrominance Cb.

For another broad aspect, when the noise type is different from the fivecited cases and when equivalent additive noise variance calculation ispossible, the equivalent additive noise variance calculator 208 shall bemodified but shape adaptive window mean or variance estimation principlewill remain.

Now referring to FIG. 3, there is illustrated an embodiment of anembedded motion estimation temporal recursive filter (EME-TRF).

The embedded motion estimation temporal recursive filter may bedecomposed into two parts 300 and 350. The first part 300 comprises asimple temporal recursive first order filter receiving the noisy videoinput signal g 101 to provide {circumflex over (f)} a temporallyfiltered video signal 107. Following the signal direction, it can beseen that the adder 301, multiplier 304, adder 306, frame delay 307 andsubtraction operator 309 constitute the temporal recursive filtergoverned by the following:{circumflex over (f)}=(g−(t−{circumflex over (m)}))·b ₀+(t−{circumflexover (m)})  (5)

in which b0 is filter coefficient signal 312, t is the frame delayversion signal 308 of the filtered image {circumflex over (f)},{circumflex over (m)} is an estimated motion signal 352 and(t−{circumflex over (m)}) is the filter signal feedback 302. The filtersignal feedback 302 may be interpreted later as motion-compensatedprevious filtered image.

The second part 350 of the embedded motion estimation temporal recursivefilter receives four input signals, i.e. the noisy video input signal101, the equivalent additive noise variance 105, the noise type signal110 and finally from the first part TRF, a frame delay version t signal308 of the filtered output video {circumflex over (f)}. Two maincharacteristics of the second part are the use of the criterion ofminimum noise variance (MNV) and embedded motion estimation (EME)technique described in detail below for each pixel of the processedimage.

It is worthwhile to mention that the proposed embedded motion estimationis complementary to classical motion estimation and compensation.Referring now to FIG. 4, classical motion estimation 413 and motioncompensation 415 are now incorporated to the temporal recursive firstorder filter part 400. The classical motion estimation 413 receives thenoisy video input signal 101 and the previous filtered image t signal308 and provides motion vector signal 414. The skilled addressee willappreciate that a discussion of classical motion estimation techniquesis not required here. The motion vector signal 414 is provided now tothe motion compensation 415 to provide motion-compensated signal tc 416.Similar to the previous FIG. 3, the motion-compensated signal tc 416 isapplied to a positive input of a subtraction operator 309 for and alsoto calculator 315. The former subtraction operator 309 provides embeddedmotion estimation recursive filter feedback signal (t_(c)−{circumflexover (m)}) 302. The calculator 315 performs proposed embedded motionestimation and temporal filter coefficient calculation.

Before the description of the embedded motion estimation and temporalfilter coefficient calculator 315, it is interesting to introduce sometheoretical background. In the following, the embedded motion estimationconcept and minimum noise variance criterion is discussed.

Embedded Motion Estimation Concept

g is the input noisy image with noise variance σ_(n) ², {circumflex over(f)} the present filtered output image with noise varianceσ_({circumflex over (n)}) ² and t is the past filtered image. Withoutclassical motion estimation and compensation, the recursive filterfeedback signal is the difference (t−{circumflex over (m)}), where{circumflex over (m)} is a local estimated motion value which will bedescribed later in more detail. With classical motion estimation andcompensation, previous filtered image t will be simply substituted byits motion compensated version tc. However, for simplicity purpose, itshould not be necessary to discuss further the case where there is thepresence of classical motion estimation and compensation.

It is further assumed that the zero mean component of the input randomnoise in each frame is independent of the image signal. At each pixel(c,r), the motion and noisy images are defined as:

$\begin{matrix}\left\{ \begin{matrix}{m = {f_{- 1} - f}} \\{g = {f + n}} \\{t = {{f_{- 1} + {\hat{n}}_{- 1}} = {f + m + {\hat{n}}_{- 1}}}} \\{\hat{f} = {f + \hat{n}}}\end{matrix} \right. & (5)\end{matrix}$

In this expression, f and f⁻¹ represent the original and the pastoriginal frames respectively. The variable m is referred to as themotion value and is defined as the difference (f⁻¹−f) between the twooriginal image amplitudes. The parameters n and {circumflex over (n)}⁻¹represent the noise in the present frame and the past filtered frame.Finally, {circumflex over (n)} is the residual output noise in thepresent filtered video frame. These above mentioned noises are zeromean.

For additive noise, there are reasons for using the additive motionvalue m:

If {circumflex over (m)} is an estimate of motion m, then the difference(t−{circumflex over (m)}) can be interpreted as a motion compensatedimage. From the equation of the filtered signal output:{circumflex over (f)}=b ₀ ·g+(1−b ₀)·(t−{circumflex over (m)}),  (6)

The residual output noise may be shown as a function of noise and motionbut free of image signals:{circumflex over (n)}=b ₀ ·n+(1−b ₀)·({circumflex over (n)} ⁻¹+m−{circumflex over (m)})  (7)

Moreover, the definition of pixel-based motion value m in Equation (5)is different from the usual motion models which are not, in turn, alwayssuitable for real life applications. The use of the proposed motionvalue definition of m is still plausible even when a scene changeoccurs. Finally, a similar multiplicative motion value definition whichcorresponds to the image ratio could be used for multiplicative noise.

Since the estimated motion value is required for the development of thenoise reducer, the proposed technique is called “Embedded MotionEstimation” (EME). Unlike the costly temporal approach for a nonlinearfiltering (see A. K. Katsaggelos, R. P. Kleihorst, S. N. Efstratiadisand R. L. Lagendijk, “Adaptive Image Sequence Noise Filtering Methods”,SPIE Vol. 1606 Visual Communication and Image Processing 1991, pp716-727), the motion information in the present document will beextracted mainly in the spatial domain.

Considering a sliding window surrounding the pixel of interest; with themodel in equation (5), local value of m may be estimated through mean ofthe difference image (t−g) in the processed window. In order to get someprecision for mean estimation, pixels of relative coordinates (i,j) inthe window may be classified in two sets i.e. similar or not to theconsidered pixel of coordinates (c,r). This is basically the shapeadaptive window technique which has already been presented in equation(2) of a previous module.

A possible local estimate {circumflex over (m)}(c,r) of the motion valuemay then be obtained by the following weighted local mean:

$\begin{matrix}\begin{matrix}\begin{matrix}{{\hat{m}\left( {c,r} \right)} = {E\left\{ {{t\left( {c,r} \right)} - {g\left( {c,r} \right)}} \right\}}} \\{{\hat{m}\left( {c,r} \right)} = {\frac{1}{N\left( {c,r} \right)}{\sum\limits_{i}{\sum\limits_{j}\left( {\left( {{t\left( {c,r} \right)} - {g\left( {c,r} \right)}} \right) \cdot {\omega_{ij}\left( {c,r} \right)}} \right)}}}}\end{matrix} \\{{N\left( {c,r} \right)} = {\sum\limits_{i}{\sum\limits_{j}{\omega_{ij}\left( {c,r} \right)}}}}\end{matrix} & (8)\end{matrix}$

where N(c,r) is the number of chosen pixels in the window around thepixel of interest at (c,r).

Moreover, in order to reduce low frequency residual noise artifacts instill parts of the image, the estimated motion value {circumflex over(m)} can be further refined as:

$\begin{matrix}{{\hat{m}\left( {c,r} \right)} = \left\{ \begin{matrix}{{\hat{m}\left( {c,r} \right)},{{{if}\mspace{14mu}{{\hat{m}\left( {c,r} \right)}}} \geq {T2}}} \\{0,{otherwise}}\end{matrix} \right.} & (9)\end{matrix}$

Substituting the definitions of g and t in equation (5) and assumingthat m is a constant value for the chosen pixels in the window, yieldsthe following expressions:

$\begin{matrix}{{\hat{m}\left( {c,r} \right)} = {\frac{1}{N\left( {c,r} \right)}{\sum\limits_{i}{\sum\limits_{j}{\left( {{m\left( {c,r} \right)} + {{\hat{n}}_{- 1}\left( {c,r} \right)} - {n\left( {c,r} \right)}} \right){\omega_{ij}\left( {c,r} \right)}}}}}} & (10) \\{{\hat{m}\left( {c,r} \right)} = {{m\left( {c,r} \right)} + {\frac{1}{N\left( {c,r} \right)}{\sum\limits_{i}{\sum\limits_{j}{\left( {{{\hat{n}}_{- 1}\left( {c,r} \right)} - {n\left( {c,r} \right)}} \right){\omega_{ij}\left( {c,r} \right)}}}}}}} & (11)\end{matrix}$

Finally, from equation (11), it can be shown further that:E{{circumflex over (m)}(c,r)}=m(c,r)  (12a)

and

$\begin{matrix}{{{var}\left( \hat{m} \right)} = {{\frac{1}{N\left( {c,r} \right)}{\sum\limits_{i}{\sum\limits_{j}{\left( {{{var}\left( {\hat{n}}_{- 1} \right)} + {{var}(n)}} \right){\omega_{ij}\left( {c,r} \right)}}}}} \leq {2\sigma_{n}^{2}}}} & \left( {12b} \right)\end{matrix}$

In other words, the estimated value {circumflex over (m)} in equation(10) is unbiased with finite variance. Thus, it can be used for motioncompensation in a temporal filter at each pixel of coordinates (c,r):t−{circumflex over (m)}=f ⁻¹ +{circumflex over (n)} ⁻¹ −{circumflex over(m)}=f+m+{circumflex over (n)} ⁻¹ −{circumflex over (m)}≅f−{circumflexover (n)} ⁻¹  (13)

Temporal Recursive Filter and Minimum Residual Noise Variance Criterion:

The residual noise at the temporal filter output is given by equation(7). At the considered pixel of coordinates (c,r), an equivalent noisen1 is defined and is composed of past filtered noise and of an unbiasedestimate of motion value:n ₁ ={circumflex over (n)} ⁻¹ +m−{circumflex over (m)}  (14)

From the equations (7) and (14), the variance of residual filtered noiseoutput may be calculated by the following equation:σ_({circumflex over (n)}) ² =b ₀ ²·σ_(n) ²+(1−b ₀)²σ_(n) ₁ ²+2b ₀(1−b₀)cov_(n,n) ₁   (15)

In order to optimize the performance, minimum noise variance criterionis used in the present filter. It can be shown that the filtercoefficient b0 may be determined as:

$\begin{matrix}{b_{0} = {\max\left( {\frac{\sigma_{n_{1}} - {cov}_{n,n_{1}}}{\sigma_{n}^{2} + {\sigma_{n_{1}}^{2}2{cov}_{n,n_{1}}}},ɛ} \right)}} & (16)\end{matrix}$

where ε is a small offset value such as ⅛ or 1/16 in order to overcomeeventual excessive precision. There are two remaining unknown values tobe determined in this equation i.e. the equivalent noise variance σ_(n)₁ ² and the covariance cov_(n,n) ₁ .

From the definition of cov_(n,n) ₁ =E{n·n₁}−E{n}·E{n₁} and the followingexpression of the equivalent noise n₁:

$\begin{matrix}{n_{1} = {{m + {\hat{n}}_{- 1} - \hat{m}} = {{\hat{n}}_{- 1} + {\frac{1}{N} \cdot {\sum\limits_{i}{\sum\limits_{j}{\left( {{n\left( {i,j} \right)} - {\hat{n}\left( {i,j} \right)}} \right) \cdot \omega_{ij}}}}}}}} & (17)\end{matrix}$

it can be shown that:cov_(n,n) ₁ =σ_(n) ² /N  (18)

The variance σ_(n) ₁ ² of equivalent noise will be calculated with thefollowing term h defined as:h=n−n ₁ =g−t+{circumflex over (m)}=(n−m)−({circumflex over (n)} ⁻¹−{circumflex over (m)})  (19)

The term h is a random process of zero mean. With further calculations,one obtains:

$\begin{matrix}{{\sigma_{h}^{2} = \left\lbrack {{\left( {1 - \frac{2}{N}} \right)\sigma_{n}^{2}} + \sigma_{n_{1}}^{2}} \right\rbrack}{or}{\sigma_{n_{1}}^{2} = {\sigma_{h}^{2} - {\left( {1 - \frac{2}{N}} \right)\sigma_{n}^{2}}}}} & (20)\end{matrix}$

The local σ_(h) ² may be estimated directly from (t−g−{circumflex over(m)}) with the previously determined shape adaptive window surroundingthe considered pixel at (r,c):

$\begin{matrix}{\sigma_{h}^{2} = {\frac{1}{N}{\sum\limits_{i}\;{\sum\limits_{j}\;{\left( {t_{ij} - g_{ij} - {\hat{m}}_{ij}} \right)^{2}\omega_{ij}}}}}} & (21)\end{matrix}$

The filter coefficient b0 given in equation (16) becomes:

$\begin{matrix}{b_{0} = {\max\left( {\frac{\sigma_{h}^{2} - {\sigma_{n}^{2}\left( {1 - \left( \frac{1}{N} \right)} \right)}}{\sigma_{h}^{2}},ɛ} \right)}} & (22)\end{matrix}$

In the proposed implementation, the term (1/N) is omitted, i.e.cov_(n,n) ₁ =0, σ² _(h) is weighted by a factor C equals to 0.75 for Yand 1 for Cr and Cb components and ε is set to be (⅛):

$\begin{matrix}{b_{0} = {\max\left( {\frac{{C\;\sigma_{h}^{2}} - \sigma_{n}^{2}}{C\;\sigma_{h}^{2}},\frac{1}{8}} \right)}} & (23)\end{matrix}$

Moreover when Cσ² _(h)<σ² _(n), in order to reduce excessive filtering,the filter coefficient b0 is set equal to an empirical value 15/64:

$\begin{matrix}{b_{0} = \left\{ \begin{matrix}{{\max\left( {\frac{{C\;\sigma_{h}^{2}} - \sigma_{n}^{2}}{C\;\sigma_{h}^{2}},\frac{1}{8}} \right)},\mspace{14mu}{{{if}\mspace{14mu} C\;\sigma_{h}^{2}} > \sigma_{n}^{2}}} \\{{15\text{/}64},\mspace{79mu}{{else}.}}\end{matrix} \right.} & (24)\end{matrix}$

The residual output noise variance required for further spatialfiltering is equal toσ_({circumflex over (n)}) ² =G·σ _(n) ² ·b ₀.  (25)

in which the empirical factor G is equal to:

$\begin{matrix}{G = \left\{ \begin{matrix}{1,} & {{for}\mspace{14mu}{Additive}\mspace{11mu}({N1})\mspace{14mu}{or}\mspace{14mu}{Multiplicative}\mspace{14mu}({N2})\mspace{14mu}{Noise}} \\{\frac{1}{2},} & {{else}.}\end{matrix} \right.} & (26)\end{matrix}$

It is worthwhile to mention that minimum noise variance is alwayspossible when local input noise power is known. In another word, theminimum noise variance is not restricted to fixed or varying local inputnoise power.

Referring now to FIG. 4, there is illustrated an embedded motionestimation and temporal recursive filter coefficient calculator. FIG. 4represents hardware or software implementation of the above theoreticalsection.

The embedded motion estimation and temporal recursive filter coefficientcalculator 351 receives four signals i.e. the noisy video input signal g101, the noise variance signal 105, the noise type signal 110 anddepending on the case, the previous filtered signal t 308 or theclassical motion compensated previous signal tc 416.

The previous filtered signal t 308 and the noisy video input signal g101 are applied together to the subtraction operator 501 (as shown inFIG. 5) to form the difference (t−g) 502 required for estimated motion{circumflex over (m)}. The difference signal 502 is provided to the lowpass filter 503, to the shape adaptive window mean estimator 509 and tothe positive input of subtraction operator 518.

The low pass filter 503, part of shape adaptive window technique, isused for robustness against noise. Its proposed impulse response isgiven again as follows:

${{lp}\; 503\left( {c,r} \right)} = {\begin{bmatrix}7 & 7 & 7 \\7 & 8 & 7 \\7 & 7 & 7\end{bmatrix}\text{/}64.}$

Low pass filter output 504 is provided to shape adaptive window 506which provides in turn local binary classification results ω_(ij)(c,r)507 and their corresponding total number N(c,r) 508. In the embodimentdisclosed, the window size is 5 lines by 11 columns. These two signalsalready described by Equations (2) and (3) are provided to the shapeadaptive window mean estimator 509 and to shape adaptive window varianceestimator 522.

The shape adaptive window mean estimator 509 receives also thedifference signal (t−g) 502 and provides an estimated motion valuesignal 510 in accordance with Equation (8). Absolute value operator 511,comparator 514 and multiplexer 517 follow the shape adaptive window meanestimator 509 as shown in FIG. 5. The multiplexer 517 provides estimatedmotion {circumflex over (m)} in accordance with Equation (9). TheMultiplexer output 352 corresponds to the final estimated motion withreduced low frequency residual noise artifacts. Estimated motion signal{circumflex over (m)} 352 is applied in turn to negative input of adders309 and 518. The adder 309 in FIG. 3 or FIG. 4 provides the feedbackdifference signal (t−{circumflex over (m)}) or (t_(c)−{circumflex over(m)}) 302 for temporal recursive filter. In accordance with Equation(19), the adder 518, shown in FIG. 5, generates signal (−h) 519 fromwhich the variance is required for filter coefficient calculation.

The variance of the signal h, σ² _(h), is also computer by shapeadaptive window technique. The signal (−h) 519 is then applied first tosquaring device 520 from which the output 521 and shape adaptive windowparameters signals 507 and 508 are applied in turn to shape adaptivewindow variance calculator 522. For practical purposes, the shapeadaptive window variance calculator 522 implements a modified version ofEquation (21) and provides Cσ² _(h) at its output 523 in which C is anempirical factor function of video components. Precisely, the output 523is given by the following equation:

$\begin{matrix}{{{C\;\sigma_{h}^{2}} = {\frac{K}{N}{\sum\limits_{i}\;{\sum\limits_{j}\;{\left( {t_{ij} - g_{ij} - {\hat{m}}_{ij}} \right)^{2}\omega_{ij}}}}}},\mspace{14mu}\begin{matrix}{C = {0.75\mspace{14mu}{for}\mspace{14mu} Y}} \\{{C = {1\mspace{14mu}{for}\mspace{14mu}{Cr}}},{Cb}}\end{matrix}} & \left( {21b} \right)\end{matrix}$

As described previously, the output 523 required for minimizing outputnoise variance, is provided to MNV Filter Coefficient Calculator 524together with the input noise power σ² _(n) 105. The MNV filtercoefficient calculator 524 determines a filter coefficient value b₀ inaccordance to Equation (23). The determined b₀ signal 525 is thenprovided to practical refiner 526.

The practical refiner 526 receives the determined b₀ signal 525, thevariance signal Cσ² _(h) 523 and the input noise power σ² _(n) 105 andmodify the filter coefficient value b₀ for some specific condition givenin Equation (24) to provide final coefficient value signal b₀ 312. Itwill be appreciated that the final coefficient value signal b₀ 312 isthe final coefficient for the temporal recursive first order filter 300shown in FIG. 3 or 400 shown in FIG. 4.

The final coefficient value signal b₀ 312, the input noise variance σ²_(n) 105 and the noise type signal 110 are provided to residual noisevariance estimator 527 to provide an estimated power 108 of residualnoise in temporally filtered video output {circumflex over (f)} 107 inaccordance to Equations (25) and (26).

The temporally filtered video {circumflex over (f)} 107 and itsassociated residual noise power signal 108 are then provided to thespatial noise reducer 109 as shown in FIG. 1.

Finally it is worthwhile to mention that the proposed embedded motionestimation and minimization of residual noise reducer works for nonrecursive structure. However, it has been contemplated that therecursive noise reducer provides better results.

Now referring to FIG. 6, there is shown an embodiment of the shapeadaptive windowing spatial noise reducer (SAW-SNR).

The shape adaptive windowing spatial noise reducer (SAW-SNR) has beendisclosed in U.S. Pat. No. 6,633,683. However, introduced bandingartifact effect in slowly varying regions and generic minimum meansquare error structure for some usual types of noise are not considered.

The spatial noise reducer module is a modified version of Lee's originalMinimum Mean Squared Error (MMSE) reduction (J. S. Lee, “Digital ImageEnhancement and Noise filtering”, IEEE Trans. on Pattern Analysis andMachine Intelligence, Vol. Pami-2, No. 2, March 1980) which can bestated as follows:

Let us define an original image f(c,r), a noisy image g(c,r) as input,g(c,r)=f(c,r)+n(c,r) and finally y(c,r) will be the filtered version. Ifthe two first order local statistics, i.e. the mean m(c,r) and thevariance σ² _(g)(c,r), of the image are known, then for additive zeromean and known variance σ² _(n)(c,r) noise, the filtered signal outputis given by:y(c,r)=m(c,r)+K(c,r)[g(c,r)−m(c,r)]  (27)whereK(c,r)=max[0, (σ² _(g)(c,r)−σ² _(n)(c,r))/σ² _(g)(c,r)].  (28)

Meanwhile, the error performance is written as:

$\;{{E\left\{ \left\lbrack {{f\left( {c,r} \right)} - {y\left( {c,r} \right)}} \right\rbrack^{2} \right\}} = \left\{ \begin{matrix}{{\sigma_{f}^{2}\left( {c,r} \right)},} & {{{if}\mspace{14mu}{\sigma_{g}^{2}\left( {c,r} \right)}} < {\sigma_{n}^{2}\left( {c,r} \right)}} & (29) \\{{\sigma_{f}^{2}\left( {c,r} \right)} \cdot {{\sigma_{n}^{2}\left( {c,r} \right)}/}} & \left\lbrack {{\sigma_{f}^{2}\left( {c,r} \right)} + {\sigma_{n}^{2}\left( {c,r} \right)}} \right\rbrack & (30)\end{matrix} \right.}$

For a single linear estimator, Lee's algorithm is perfect when m(c,r),σ² _(g)(c,r) are known and when σ² _(g)(c,r)>σ² _(n)(c,r). However, forpractical situations, the two first order local statistics m(c,r) and σ²_(g)(c,r) are unknown and need to be estimated. On the other hand, whenσ² _(g)(c,r)<σ² _(n)r(c,r), using K(c,r)=0, the small details containedin the original image will be destroyed as shown by Equation (29).

In the following descriptions of SNR, the modifications proposedcomprise three major techniques shape adaptive windowing (SAW) for localmean and variance estimation, banding effect reduction (BER) for smallsignal variance case and noise power converter incorporation for genericnoise reducer structure.

The spatial noise reducer 109, as illustrated in FIG. 1, receives thetemporally filtered video signal 107 of three components (Y, Cr, Cb) andtheir corresponding estimated residual noise powers 108 provided by thetemporal recursive noise reducer 106.

In FIG. 6, the video signal 107 is provided to four units or devicesi.e. low pass filter 601, shape adaptive windowing mean estimator 607,shape adaptive windowing variance estimator 608 and positive input ofsubtraction operator 609.

The low pass filter 601, part of shape adaptive windowing technique, isgiven by the following impulse response:

${{lp}\; 601\left( {c,r} \right)} = {\begin{bmatrix}7 & 7 & 7 \\7 & 8 & 7 \\7 & 7 & 7\end{bmatrix}\text{/}64.}$

The output of the low pass filter 601 is provided to the shape adaptivewindow 604 which provides in turn local binary classification resultsω_(ij)(c,r) 605 and their corresponding total number N(c,r) 606.

In the embodiment disclosed, the window size is 5×5. These two signalsare provided to the shape adaptive windowing mean estimator 607 and tothe shape adaptive windowing variance estimator 608 to providerespectively the following output signals 610 and 612:

$\begin{matrix}{{m\left( {c,r} \right)} = {\left( {1/{N\left( {c,r} \right)}} \right){\sum\limits_{i}{\sum\limits_{j}{{{\hat{f}}_{ij}\left( {c,r} \right)}{\omega_{ij}\left( {c,r} \right)}}}}}} & (31) \\{{\sigma_{\hat{f}}^{2}\left( {c,r} \right)} = {\left( {1/{N\left( {c,r} \right)}} \right){\sum\limits_{i}{\sum\limits_{j}{\left( {{{\hat{f}}_{ij}\left( {c,r} \right)} - {m\left( {c,r} \right)}} \right)^{2}{\omega_{ij}\left( {c,r} \right)}}}}}} & (32)\end{matrix}$

The former m(c,r) 610 is provided to negative input of adder 609 andalso to adder 617. The latter 612 is provided to the input of adaptivegain calculator 613. If K(c,r) 614 denotes the adaptive gain outputsignal of 613, then it can see that {circumflex over ({circumflex over(f)}(c,r) final filtered video output 103 is realized as the followingexpression:{circumflex over ({circumflex over (f)}(c,r)=m(c,r)+K(c,r)·({circumflexover (f)}(c,r)−m(c,r))  (33)

in accordance with the minimum mean square error filter's structure ofEquation (27). The adaptive gain K(c,r) will be described in detail inFIG. 7.

Referring to FIG. 7, there is shown an embodiment of an adaptive gaincalculator.

The adaptive gain calculator receives the local variance 612 of theinput image 107 and the residual noise variance σ² _(nS) 108 in theimage input.

The residual noise variance σ² _(nS) 108 is low passed filtered by 719to provide a smoothed version σ² _(nR) 620. This signal is combinedtogether with local signal variance σ² _({circumflex over (f)}) 612 viasubtraction operator 701, divider 703 and max selector 706 to forms again signal K1 707 mathematically given by:

$\begin{matrix}{{K_{1}\left( {c,r} \right)} = {{\max\left( {0,\frac{{\sigma_{\hat{f}}^{2}\left( {c,r} \right)} - {\sigma_{nR}^{2}\left( {c,r} \right)}}{\sigma_{\hat{f}}^{2}\left( {c,r} \right)}} \right)}.}} & (34)\end{matrix}$

Equation (34) is the standard form, i.e. Equation (28), of minimum meansquare error filtering technique.

At the same times, local signal variance σ² _({circumflex over (f)}) 612is applied to comparator 712 and to comparator 714.

The comparator 712, for luminance case, gives a binary signal bsL(c,r)output 713 in accordance to

$\begin{matrix}{{{bs}_{L}\left( {c,r} \right)} = \left\{ {\begin{matrix}{1,} & {{{if}\mspace{14mu}{\sigma_{\hat{f}}^{2}\left( {c,r} \right)}} \leq \text{Small~~Threshold}} \\{0,} & {{{if}\mspace{14mu}{\sigma_{\hat{f}}^{2}\left( {c,r} \right)}} > \text{Small~~Threshold}}\end{matrix}.} \right.} & (36)\end{matrix}$

in which small threshold value 710 is chosen equal to 1.25 in an 8-bitresolution.

For the chrominance case, small threshold value 711 is more restrictive.The comparator 714 provides a binary signal output bsC(c,r) 715 with thefollowing test:

$\begin{matrix}{{{bs}_{C}\left( {c,r} \right)} = \left\{ {\begin{matrix}{1,} & {{{if}\mspace{14mu}{\sigma_{\hat{f}}^{2}\left( {c,r} \right)}} = 0} \\{0,} & {{{if}\mspace{14mu}{\sigma_{\hat{f}}^{2}\left( {c,r} \right)}} \neq 0}\end{matrix}.} \right.} & (35)\end{matrix}$

One of the above binary signals will be selected by selector 716. Theselector output 117, denoted as be(c,r), is used, in turn, to controlselector 709 which finally provides the adaptive gain K(c,r) 614:

$\begin{matrix}{{K\left( {c,r} \right)} = \left\{ {\begin{matrix}{1,} & {{{{if}\mspace{14mu}{{be}\left( {c,r} \right)}} - 1},\text{i.e.~~signal~~variance~~very~~small}} \\{{K_{1}\left( {c,r} \right)},} & \text{if~~else}\end{matrix}.} \right.} & (36)\end{matrix}$

It will therefore be appreciated that in other words, when noisy signalvariance is very small, it is not necessary to apply the filter on thesignal.

The local adaptive gain K(c,r) signal 614 is provided to multiplier 615for completing the minimum mean square error filtering.

Referring to FIG. 8, there is shown an embodiment of a region adaptivefacet based spatial noise reducer (RAFB-SNR).

As previously mentioned, in order to exploit the MMSE criterion inspatial filtering, it is necessary to know the local signal mean andvariance values with some precision. The proposed SAW for mean andvariance estimation cannot be necessarily precise when the originalsignal is not constant but slowly varying and locally represented as asloped facet (piecewise linear model) or a piecewise quadratic model. Inorder to select linear versus quadratic model, a simple imagesegmentation is required. The piecewise linear model is applied for flatregions, piecewise quadratic otherwise. Image segmentation is thususeful for an adaptation of facet model order determination.

The estimated mean value is used for de-noising signal value. In thefollowing, a region adaptive facet based spatial noise reducer(RAFB-SNR) is proposed. The spatial noise reducer comprises twodifferent innovations: a)—MMSE de-noising technique, b)—Facet models(piecewise linear or piecewise quadratic) adaptation depending onsegmented regions.

RAFB-SNR 111 as illustrated in FIG. 8 receives residual noisy threecomponent (Y, Cr, Cb) video 107 provided from TF 106 and theirscorresponding estimated noise powers 108.

The received video 107 is applied to Adaptive Facet ParametersCalculator 803, Adder 810, Facet based Local Variance Estimator 806 andImage Segmentation module 801.

Image Segmentation module 801 provides a binary signal output 802Flat/No-Flat regions. For skill people in the art, Flat/No-Flatsegmentation can be provided by a plurality of possible processingtechniques. Flat/No-Flat regions signal 802 is sent to Adaptive FacetParameters Calculator 803.

Facet Parameters utilized in the invention are the coefficients b₀(c,r),b₁(c, r), . . . , b₅(c,r) which approximate in the least squared fit anincoming signal y(i,j; c,r) in a window centered at the coordinates(c,r):

${b_{k}\left( {c,r} \right)} = {\arg\;\min{\sum\limits_{i}\;{\sum\limits_{j}\left\{ {{y\left( {i,{j;c},r} \right)} - \left\lbrack {{b_{0}\left( {c,r} \right)} + {{b_{1}\left( {c,r} \right)}i} + {\left. \quad{{{\quad\quad}{b_{2}\left( {c,r} \right)}j} + {{b_{3}\left( {c,r} \right)}i^{2}} + \left. \quad{{{b_{4}\left( {c,r} \right)}{ij}} + {{b_{5}\left( {c,r} \right)}j^{2}}} \right\rbrack} \right\}^{2}.}} \right.} \right.}}}$

This expression is utilized when the central pixel is classified tobelong to a No-Flat region. For an estimated Flat region signal, thecoefficients b₃, b₄ and b₅ are set to be equal to zero.

The coefficient b₀(c,r) corresponds furthermore to the local signal meansignal 1004:mean(c,r)=b ₀(c,r).

The six (6) coefficients b_(k)(c,r) 804 and 805 are send to Facet basedLocal variance estimator 1006 which provides in turn variance signal 807by the following expression:

$\begin{matrix}{{{var}\left( {c,r} \right)} = {\sum\limits_{i}{\sum\limits_{j}\left\{ {{y\left( {i,{j;c},r} \right)} - \left\lbrack {{b_{0}\left( {c,r} \right)} + {{b_{1}\left( {c,r} \right)}i} +}\mspace{236mu} \right.} \right.}}} \\{\left. \left. \mspace{320mu}{{{b_{2}\left( {c,r} \right)}j} + {{b_{3}\left( {c,r} \right)}i^{2}} + {{b_{4}\left( {c,r} \right)}{ij}} + {{b_{5}\left( {c,r} \right)}j^{2}}} \right\rbrack \right\}^{2}.}\end{matrix}$

Local variance signal 807 and Residual noise power σ² _(nS) 108 are usedtogether by Adaptive Gain Calculator 808 which yields in turn a gainsignal K 809. Adaptive Gain K Calculator according to MMSE criterion isas previously described.

Adder 810, Multiplier 812 and Adder 814 are utilized to form MMSEde-noising signal output 103.

Referring now to FIG. 9, there is illustrated a second embodiment of amultiple noise type spatial noise reduction (MT-SNR) apparatus.

For economical purpose or for the case of single image, the temporalfilter may be removed. Only a generic spatial noise reducer is thusrequired in such applications. The multiple noise type spatial noisereduction (MT-SNR) in FIG. 9 is proposed for the given five types ofnoise.

The multiple noise type spatial noise reduction (MT-SNR) comprises twomain blocks i.e. a noise power converter 104 and a spatial noise reducer(SNR) 109. Noise power converter 104 already described remainsunchanged. The SNR 109, realized with either SAW-SNR or RAFB-SNR, isalso unchanged except its inputs (107) and (108) accept now 101 videoinput and 105 estimated equivalent additive noise power signalsrespectively.

As an example, let us consider the multiplicative noise N2 case in whichproposed equivalent additive noise power is σ² _(n)=μ²·σ² _(u).Substituting the result into the fundamental part, basic for minimummean square error, of Equation 28 yields:

$\begin{matrix}{{K\left( {c,r} \right)} = \frac{{\sigma_{g}^{2}\left( {c,r} \right)} - \sigma_{n}^{2}}{\sigma_{g}^{2}\left( {c,r} \right)}} \\{= {\frac{{\sigma_{g}^{2}\left( {c,r} \right)} - {\mu^{2}\sigma_{u}^{2}}}{\sigma_{g}^{2}\left( {c,r} \right)}.}}\end{matrix}$

Since σ² _(u) is generally very small to compare with 1, the result iscomparable with a direct linear minimum mean square error optimizationone:

${K\left( {c,r} \right)} = {\frac{{\sigma_{g}^{2}\left( {c,r} \right)} - {\mu^{2}\sigma_{u}^{2}}}{{\sigma_{g}^{2}\left( {c,r} \right)}\left( {1 + \sigma_{u}^{2}} \right)}.}$

While illustrated in the block diagrams as groups of discrete componentscommunicating with each other via distinct data signal connections, itwill be understood by those skilled in the art that the preferredembodiments are provided by a combination of hardware and softwarecomponents, with some components being implemented by a given functionor operation of a hardware or software system, and many of the datapaths illustrated being implemented by data communication within acomputer application or operating system. The structure illustrated isthus provided for efficiency of teaching the present preferredembodiment.

It should be noted that the present invention can be carried out as amethod, can be embodied in a system, a computer readable medium or anelectrical or electro-magnetic signal.

The embodiment(s) of the invention described above is(are) intended tobe exemplary only. The scope of the invention is therefore intended tobe limited solely by the scope of the appended claims.

1. An apparatus for reducing multiple noise types in a video inputsignal, said apparatus comprising: a noise power converter for receivingand using said video input signal, a variance of additive noise, avariance of multiplicative noise and an indication of a type of noisefor estimating an equivalent additive noise variance signal; a temporalrecursive filter using said equivalent additive noise variance and saidvideo input signal for generating a temporally filtered video signal anda residual noise variance signal; a spatial noise reducer using saidresidual noise variance signal and said temporally filtered video signalfor spatially filtering said video input signal to provide a videooutput signal having reduced noise; and wherein said noise powerconverter comprises a low pass filter for filtering said video inputsignal to provide a low pass filtered signal; a shape adaptive windowfor receiving and using said low pass filtered signal, and a giventhreshold tolerance for calculating a binary signal ωij(c,r)representing local classification results within said thresholdtolerance and a signal N(c,r) indicative of similar pixels in saidwindow; a shape adaptive window mean estimator using said binary signalωij(c,r), said N(c,r) signal and said video input signal for calculatinga local estimated means signal value μY(c,r) given by${{\mu_{Y}\left( {c,r} \right)} = {\left( {1/{N\left( {c,r} \right)}} \right){\sum\limits_{i}{\sum\limits_{j}{{Y_{ij}\left( {c,r} \right)}{\omega_{ij}\left( {c,r} \right)}}}}}};$and an equivalent additive noise variance calculator using said localestimated means signal value μY(c,r), said variance of additive noise,said variance of multiplicative noise and said indication of a type ofnoise for calculating said equivalent additive noise variance signal. 2.The apparatus as claimed in claim 1, wherein said multiple noise typescomprise additive (N1), multiplicative with negative gamma γphotographic density (N2), multiplicative with positive gamma γphotographic density (N3), speckle with negative gamma γ photographicdensity (N4) and speckle with positive gamma γ photographic density(N5).
 3. The apparatus as claimed in claim 1, wherein said low passfilter has an input response as follows:${{lp201}\left( {c,r} \right)} = {\begin{bmatrix}7 & 7 & 7 \\7 & 8 & 7 \\7 & 7 & 7\end{bmatrix}/64.}$
 4. The apparatus as claimed in claim 1, wherein saidbinary signal ωij(c,r) is given by:${\omega_{ij}\left( {c,r} \right)} = \left\{ \begin{matrix}{1,{{{if}\mspace{14mu}{{{{lp}\left( {Y_{ij}\left( {c,r} \right)} \right)} - {{lp}\left( {Y_{00}\left( {c,r} \right)} \right)}}}} < {Threshold}}} \\{0,{else}}\end{matrix} \right.$ where lp(Y(c,r)) comprises said low pass filteredsignal for luminance component Y at (c,r).
 5. The apparatus as claimedin claim 1, wherein signal N(c,r) indicative of similar pixels in saidwindow is given by:N(c,r)=ΣΣωij(c,r).
 6. The apparatus as claimed in claim 1, wherein saidtemporal recursive filter comprises: an embedded motion estimation andtemporal filter coefficient calculation unit for receiving and usingsaid video input signal, said equivalent additive noise variance and apast temporally filtered video signal t for calculating said residualnoise variance signal, an estimated motion signal {circumflex over (m)}and a filter coefficient b0; and a temporal recursive first order filterfor filtering using said video input signal, said filter coefficient b0and said estimated motion signal {circumflex over (m)} to provide saidtemporally filtered video signal.
 7. The apparatus as claimed in claim6, wherein said temporally filtered video signal {circumflex over (f)}is given by:{circumflex over (f)}=(g−(t−{circumflex over (m)}))·b0+(t−{circumflexover (m)}).
 8. A method for reducing multiple noise types in a videoinput signal, said method comprising: estimating an equivalent additivenoise variance signal using said video input signal, a variance ofadditive noise, a variance of multiplicative noise and an indication ofa type of noise; temporally filtering said video input signal videosignal generating a residual noise variance signal using said equivalentadditive noise variance and said video input signal; spatially filteringsaid temporally filtered video signal using said residual noise variancesignal and said video input signal to provide a video output signalhaving reduced noise; and wherein said estimating of an equivalentadditive noise variance signal comprises low-pass filtering said videoinput signal to provide a low-pass filtered signal; calculating a binarysignal ωij(c,r) representing local classification results within saidthreshold tolerance and a signal N(c,r) indicative of similar pixels insaid window using said low pass filtered signal, and a given thresholdtolerance; calculating a local estimated means signal value μY(c,r)using said binary signal ωij(c,r), said N(c,r) signal and said videoinput signal calculating a local estimated means signal value μY(c,r)using${\mu_{Y}\left( {c,r} \right)} = {\left( {1/{N\left( {c,r} \right)}} \right){\sum\limits_{i}{\sum\limits_{j}{{Y_{ij}\left( {c,r} \right)}{\omega_{ij}\left( {c,r} \right)}}}}}$and calculating said equivalent additive noise variance signal usingsaid local estimated means signal value μY(c,r), said variance ofadditive noise, said variance of multiplicative noise and saidindication of a type of noise.
 9. The method as claimed in claim 8,wherein said multiple noise types comprise additive (N1), multiplicativewith negative gamma γ photographic density (N2), multiplicative withpositive gamma γ photographic density (N3), speckle with negative gammaγ photographic density (N4) and speckle with positive gamma γphotographic density (N5).
 10. The method as claimed in claim 8, whereinsaid low pass filtering has an input response as follows:${{lp}\; 201\left( {c,r} \right)} = {\begin{bmatrix}7 & 7 & 7 \\7 & 8 & 7 \\7 & 7 & 7\end{bmatrix}/64.}$
 11. The method as claimed in claim 8, wherein saidbinary signal ωij(c,r) is given by: $\left\{ {\begin{matrix}{1,\;{{\text{if~~}{{{{lp}\left( {{Yij}\left( {c,r} \right)} \right)} - {{lp}\left( {Y\; 00\left( {c,r} \right)} \right)}}}} < \text{Threshold}}} \\{{{\omega\;{{ij}\left( {c,r} \right)}} = 0},\;\text{else}}\end{matrix}\quad} \right.$ where lp(Y(c,r)) comprises said low passfiltered signal for luminance component Y at (c,r).
 12. The method asclaimed in claim 8, wherein said signal N(c,r) indicative of similarpixels in said window is given by:N(c,r)=ΣΣωij(c,r).
 13. The method as claimed in claim 8, wherein saidgenerating a residual noise variance signal comprises using said videoinput signal, said equivalent additive noise variance and a pasttemporally filtered video signal t for calculating said residual noisevariance signal, an estimated motion signal {circumflex over (m)} and afilter coefficient b0; and further wherein said temporally filteringsaid video input signal comprises using said filter coefficient b0 andsaid estimated motion signal {circumflex over (m)} to provide saidtemporally filtered video signal.
 14. The method as claimed in claim 13,wherein said temporally filtered video signal {circumflex over (f)} isgiven by:{circumflex over (f)}=(g−(t−{circumflex over (m)}))·b0+(t−{circumflexover (m)}).